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ABSTRACT 

m«aanr.<«« Two Of the fflost popular assessment instruments for 

Sut^o2 Lf ^^'""''""^ educational outcomes are the college 

le^lTe ^S) ^LlnL TT'-'' ''^"^''•^ °' Educational TesLng 
se^itivlS f °' ^''^^^ ^^"'^ compared, and the 

eva!iltl^ in .hf ^ T l"" students' educational experiences was 
ylai s^Ior^ T.t t 'Sorters of the 1987-88 school 

llllL li t Univers-.y of Tennessee (Knoxvllle) took one of 

these tests. Overall, more seniors took the COKP (n=l 261) than ^h! 
e^r::tL''°'''' T^rty-elght senloS toik litf^ 

^ire^Sft Irl ^^^^^ unldlmensionai 

measures that are not particularly sensitive to students' aenerai 
education experiences. Both tests represented measures of !nSmdual 
differences and were powerful measures of academic Siilty^Su^ 

li^lil Profl^r'' '° inciiviaual differences, *,otf the^Mf ^d tne 
Academic Profile are not considered aiiproprlate for evaluatina f h*. 

S'Lvfirr^' P-ograms.institutLnrwo'S ^ot L 

the^^nin!«. subscaies of either examination to evaluate 

the components of a curriculum. Six tables give study results 



« Reproductions supplied by EDRS are the l^st that can I 

from the original document. 
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A COMPARISON OF THE COLLEGE OUTCfflflS MEASURES PROGRAM (COMP) EXAM 

AND THE ETS ACADEMIC PROFILE 

Gary R. Pike 
Associate Director 
AAsessaent Resource Center 



During the 19808, the reports of national advisory cows iss Ions, coupled 
with the actions of state higher education conslssions and regional accrediting 
associations, have sparked intens*? Interest in the assessaent of student educa- 
tional outcoses (El-Khawas, 1987; Ewell. 1987). Responding to the heightened 
Interest in student learning, test developers have Introduced several nei« 
outcoaes measures, nany designed to assess the effectiveness of general educa- 
tion prograas (Pike, in press). 

Because so «any assessnent instrujsents are available, it is critical that 
institutions utilize outcoaes Measures that accurately reflect their curricula. 
Missions, and student characteristics. Judging the appropriateness of an 
assessaent Instruaent requires that the content, psychoaetric properties, and 
construct validity of the instruaent be carefully exaained. 

Two of the Bost popular assessaent Instruaents are the College Outcoae 
Measures Prograa exaainatlon (Aaerlcan College Testing Prograa. 1987) and the 
ETS Acadealc Profile (ETS College and University Prograas, 1988). Both the 
COMP exaa and the Acadealc Profile are designed *o aeasure the outcoaes of 
general education. Because it is difficult, if not lapossible. to identify a 
core of knowledge (content) that is coaaon to general education prograas at 
aost colleges and universities, both the COMP exaa and the Acadealc Profile 
alnlalze the need to recall specific facts. However, according to staff at ACT 
and ETS. faalliarlty with content laproves test perforaance. 

The objective fora of the COMP exaa takes approxiaately 2 1/2 hours to 
adalnister and contains 60 questions, each with two correct answers. These 
questions are divided aaong 15 separately tiaed activities drawing on aaterials 
(stlauli) from television prograas, radio broadcasts, and print aedla. Stu- 
dents taking the COMP exaa are Instructed that thei is a penalty for guessing 
(i.e.. Incorrect answers will be subtracted froa st dents' scores). The coabl- 
nation of two correct answers for each question and the guessing penalty aeans 
that each question on the COMP exaa is worth 4 points. The aaxiaua possible 
score on the COMP exaa is 240 points. 

In addition to a total score, the COMP exaa provides three content 
subscoies (Functioning within Social Institutions [FSI], Using Science and 
Technology fuSTJ, and Using the Arts [UA]) and three process subscores (Coaau- 
nlcatlng |C0M], Solving Probleas [SPj, and Clarifying Values fCVj). It is 
difficult to deteralne precisely what underlying constructs these scales are 
designed to aeasure because the technical aanual for the COMP exaa only pro- 
vides one-paragraph descriptions of the subscales (Forrest & Steele, 1982). 

The three-hour version of the Acadealc Profile consists of 144 questions 
and 18 designed to provide scores for individuals. The aaxiaua possible score 
on the Academic Profile Is 144 points. The questions on the Acadealc Profile 
are designed to aeasure four skills (Reading [READJ , Writing [WRIT], Critical 
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Imrll'!*"! ^?]- "l*^ Mathe.atlc» [MATH]) across three content areas (Huwnltles 
[HUM], Social Sciences {SSJ. and Natural Sciences fNSj). As Is the case with 
the COMP exa«. ETS provides little inforaatlon about the constructs the seven 
subscales of the Acadealc Profile are designed to aeasure. Students taking the 
Academic Profile are told that there is no penalty for guessing, Like the COMP 
exaa. the Academic Profile uses aultiple stlauli. However, all of the stlauli 
in the Acadeaic Profile consist of written passages. 

AS part of its perforaance funding effort, the University of Tennessee - 
Knoxvllle has begun a study to coapare the COMP exaa and the Acadeaic Profile. 

!JnL t < " 'ff?''* ^^'^'^ research efforts. This report is organized 

around two topics: (1) a coaparison of the characteristics of each test, and 
(2) an evaluation of the sensitivity of each test to students' educational 
experiences, m reading this report, it is important to remember that these 
evaluations represent only one part of an ongoing effort to analyze a variety 
of assessment instruments. *icvy 



Procedures 

I.... I^^.J^^^nffao^^" gathered during the Fall and Winter quar- 

ters or cne i»e7-»8 academic year. During the Fall quarter. 1064 seniors were 
tested using either the objective fora of the COMP exaa or the three-hour 
nSfl °?H-„^ f Profile. Slightly aore students took the COMP exam 

(598) than the Acadeaic Profile (466). During the Winter quarter. 1007 stu- 

Ac^"de%rc"p^o?ilT(34^r'"' '''' '''''' '''' 

Assignaent to a testing group was based on two criteria: First all 
students who had taken the COMP exam as freshaen were assigned to thi COMP 
testing group as seniors. Stcond. students who bad not been tested as freshaen 
were randomly assigned to either the COMP or the Acadeaic Prof lie tesUng 

.nH ^8^1"^^^! Winter quarter. 38 seniors agreed to take both the COMP exaa 
and the Academic Profile. All of these students were volunteers, and they were 
ir?hf,^'.!« '"""'^ participation. Despite the fact that all of the students 
tL i^nLpr!?. '^^^ selected, their entering ability levels were close to 

the University average. 

Cowparing the characteristics of the COMP exaa and the Acadeaic Profile 
involved examining the psychometric properties of the tests, both independently 

ance on"the t^oT . '''h' '""'llV descriptions of student ' 

ance on the two tests and provided measures of the interrelationships aaong 
subscales. By exaaining these interrelationships, it is possible to draw 
inferences about what Is being Measured by the two tests. 

D ^fM evaluate the sensitivity of the COMP exaa and the Acadeaic 

Profile to students' educational experiences, the scores of UTK students on the 
^re der!.:rfr«"'''"'!^ "'l' course-taking. Coursework aeasurea 

^ok roiip p^L ^^^i^^^f/^fPl^-^ntal Iteas students coapleted when they 

course-L^ 2lf?p^!/^^^Lf '"^^ the questions on 

«»n,^^t J ^^ff«^*'<» slightly depending on the test, the <iuestlons aeasured the 

amount of coursework in four areas: huaanities. social sciences, matheaatlci 
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and natural sciences. For students taking the Acadeoilc Profile, natural sci- 
ences coursework and aatheBatlcs coursework were combined 



Test Characteristics 



Score Profiles 



nltl presented in Table 1 describe the performance of UTX students on 
the COMP exaa and the Academic Profile. Test data include mean scores (X) and 
■ean percentage correct scores {K%) for each Instrument over the two testina 
periods. In order to make scores^n the two tests comparable, the mean per- 
centage co£Ejct for norm groups (MM*) and ratios of the UTK means to these 
norms (X»/NM») also are provided. Standard deviations (Sx) are included as 
Indicators of score dispersion. 



Table 1: 

Student Performance on the COMP Exam and the Academic Profile 



HUM 

SS 

NS 

READ 
WRIT 
CT 

HATH 

TOTAL 



Fall (N-598) 



COMP EXAM 





X 


X% 


liM% 


X*/NM» 


Sx 


FSI 


63.58 


79 


78 


1.01 


5.66 


UST 


64.60 


81 


78 


1.03 


5.74 


UA 


62.59 


78 


75 


1.04 


6.32 


COM 


54,26 


75 


73 


1.03 


7.22 


SP 


77.92 


85 


78 


1.09 


5.65 


CV 


58.36 


81 


79 


1.02 


5.70 


TOTAL 


190.77 


79 


77 


1.03 


14.89 



61.35 
64.12 
60.12 
52.39 
75.44 
57. ?6 

185.65 



Fall (N«466) 



ACADEMIC PROFILE 



29.76 
28.89 
28.72 
23.03 
23.42 
19,43 
21,50 

87.38 



x» 


SM% 




Sx 


X 


62 


50 


1.24 


7.94 


28,72 


60 


50 


1.20 


7.79 


28.04 


60 


50 


1.20 


7.97 


28,27 


64 


50 


1,28 


6.25 


22.20 


65 


50 


1.30 


6,10 


22.68 


54 


50 


1,08 


6,49 


18.86 


60 


50 


1.20 


6.20 


21.28 


61 


50 


1,22 


22. 13 


85,02 



Winter (N=683) 

X* mn X%/NM» Sx 



77 
80 
75 
73 
79 
80 

77 



78 
78 
75 
73 
78 
79 

77 



.99 
1.03 
1.00 
1.00 
1.01 
1.01 



6.68 
6.48 
5.37 
7.22 
6.66 
5.41 



1.00 15.02 



Winter (N«344) 

X* Im x*/Jm Sx 



60 
58 
59 
62 
63 
52 
59 

59 



50 
50 
50 
50 
50 
50 
50 

50 



1.20 
1.16 
1.18 
1.24 
1.26 
1.04 
1.18 



8.42 
7.95 
8.03 
6.61 
6.57 
6.37 
6.19 



1.18 22,81 
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Nor., for the >»c.de.lc PrdfUe « .'2:"i:;"fr«'lS 

An exaaination of the m*»»n^ 
students perforaed beUeJ on JSJ 232p'!"r^H''""'* -"^^'ts that UTK 

scores averaged 79 percent corrector ^^SS ^" ^'^^^'^^^ Profile ^al 
correct In the Winter. By way of coSL^^ ^""^^ ^" ^'aH and 77 Derc«nt 

Winter. On the six subscalea of thJrnl^ 5® percent correct in th- 

percent correct (Co««unlcatrnJ anS sf ^/^"f* '"'^ avera^^d be ieeS 75 

winter scores averaged betweer73 peJceS? coJ^ecr^r' P^otlell^ 
cent correct (Clarifying Values and ufin^ c r^*"^ (CoMunlcatlng) and 80 per- 
seven subscales of the Jcade«!c ?rof?!e ""^ Technolog;, . Por the 

cent critical Thinking) and 65 percint V^n T?"^* averaged between S4 per- 
adiilnlstratlon. scores averaged tltllTn SS"*^'^ ^«"-ltlng). For the Wlnte? 
and 63 percent correct (Wrltfng) "^'^ ""^'^^^^ (Critical ThlnJclng, 

A comparison of noras for thp rcato ^ 
the suggestion that students perfor«^ttf** Acadeaic Profile refutes 

age correct 1.22 fo, total "ores oTth'*/''5"" Jercen ^ 

and 1.00 respectiveiy. " quarter, these ratios »ere l.is 

The data presented in Tahiw i ,1- 

tlnna f«« ♦ * . Winter quarter. For th*. rnuo I " ^n the Fall guar- 

present nonetheless. """"'e. of the two exU^ they are 

variaS?: ^hrsiororrjsMP'er r-'-'^ — 

"n.e o. the CO. T^To t'o' l^oTls'- h^" ^ - £^ 

^ OESTCOWWAlUlli 
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«Il%!!''^?n:i*' ^'-^^ ^" negative scores 

on the COMP exa« (0 to 120) are excluded, the observed differences in 
variability would be surprising. 

One possible explanation for less variability in COMP exaa scores is the 
loi* level of test difficulty for the COMP exaa. The fact that the COMP exan is 
a relatively easy test creates a ceiling effect which restricts the range of 
variation in students' scores at the top of the scale. One practical result of 
the greater variability in scores on the Acadeslc Profile is that the test is 
better able to differentiate aaong students than is the COMP exa«. 

.Hiull^ llll lu^^.^^l ^""/^ difficult, and evidences less vari- 
t^iii^^I.it? . Acade.ic Profile has important practical conse«n,ences for the 
interpretation of test scores. Low difficulty levels, coupled with low levels 
of score variance and the fact that each question on the COMP exaa can be worth 
as aany as 4 points, create a situation in which saall changes in student 
performance can have enor.ous effects on percentile ranks. For exaaole a 
Steie in students ' responses on i of the 60 Questions (2 responses) on the 
COMP exa« would produce a change In the «e5 total score of 4 ^ints (oCt o? a 
'S!ni''! 'tl ' scorrtha^STTs-l^ liJan ff^frcent 

change n the possible score, but it translates into approxlaately a 10 percen- 
tile E^lnl gain or loss for scores between the 30th and 70th percentiles Th^ 
score decline of slightly over 5 points fro. the Fall to the Sinter quaners 
for UTK seniors (see Table 1) translates into almost a 15 percentile point 
decline from the 60th to the 45th percentile. 



Relationship s Among Subscales 

Examining the relationships among the subscales of an assessment instru- 
ment can provide useful information about whether that Instrument actually 
measures the outcomes it purports to measure (Messlck. 1987). For the Durooses 
of the present investigation, the issue is whether the subscales of the'^S 
exam and the Academic Profile actually measure distinct, although possibly 
related, aspects of general education. 

r.i.nl^!/^''^* evaluating the relationships among subscales involved 

Acideiic ??ofi!r .'?h' subscales. Because both the COMP exam and the 

Academic Profile use the same questions to measure content and process/skill 
areas, correlations had to be calculated separately for the subscales within 
the content areas and within the process/skill areas. Correlations between 
content and process/skill subscales would not be statistically vaUd ind!ca- 

are orewntedlrT^h^^',*'"'"'"'"^" ?' ""^ '^'^^ ^Oe Ac.de.lc Profile 

are presented In Table 2, These correlations are based on data fro. both the 

r-l\ j,1n h ""^ '^''^"iolenu m the first subtable depict 

™ s in 'f <=°>^ ««» »>"' "ased on national data. 

Tst,iV, ToLf i!"" t"""!"! "port o» the COW exa. (Forrest 

?e,U%1 pe lod ■ "*"'='-nt» fe first subtable are the sa.e or both 
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Table 2: 

Correlations Aaong Subscores on the COMP Exaa and the Acadealc Profile 



FSI 

UST 

UA 

CON 

SP 

CV 



FSI 

UST 

UA 

CON 

SP 

CV 



HUN 

SS 

NS 

READ 
WRIT 
CT 

NATH 



Fall 



FSI 


UST 


UA 


.63 


.59 


.55 


.91 


.68 


.59 


.86 


.88 


.66 



CONP EXAN - NATIONAL SCORES 



CON SP CV 



Winter 



FSI 


UST 


UA 


.63 


.5P 


.55 


.91 


.68 


.59 


.86 


.83 


.66 



CON SP CV 



.66 


.56 


.54 


.84 


.68 


.57 


.82 


.85 


.66 



.66 


56 


.54 


.84 


.68 


.57 


.82 


.85 


.66 



Fall 



COMP EXAM - UTK SCORES 



Wlnt ir 



FSI 


UST 


UA 


COM 


SP 


CV 


FSI 


UST 


UA 


CON 


SP 


.61 


.56 


.42 








.69 


.54 


.48 






.98 


.63 


.41 








.78 


.70 


.50 






.68 


.63 


.67 








.78 


.85 


.49 












.75 


.48 


.42 








.73 


.46 








.73 


.58 


.43 








.70 


.65 








.60 


.70 


.65 








.70 


.79 



Fall 



ACADEMIC PROFILE - UTK SCORES 



Winter 



HUN 


SS 


SS 


.84 


.82 


.79 


.98 


.83 


.82 


.94 


.99 


.84 



,88 
.98 
90 



.81 

.96 
1.00 
.74 



.78 .82 
.80 .75 



,85 
,88 
93 



78 
80 
86 



,93 
80 



,81 
83 



.60 
.64 
.67 
.80 



.83 
.98 
.99 
.78 



81 
83 
93 
74 



,81 
,76 
,80 
84 



Above Diagonal = Correlations 
Diagonal = ReUabllities 

Below Diagonal = Correlations (Olsattenuated) 



CV 



.47 
.50 
.62 



READ WRIT CT MATH HUM SS NS READ WRIT CT MATH 



.63 
.60 
.67 
.79 
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The second nuht^ui ^ 
"Ubscaiet of th. omm "''••"it* data conc.rnin- ... 

reiatlon.hlps „oS?thrf : "« »«btabf« "'•""nahlp. a..„, th. 

«... .-.vs,:."-- r„-£S?:ri"m 

Coefficients above the di. seniors, 
"e based on .uDsmi; .! "efficient (Ouliikfif «l<»"ated usln, , 

n» tfte formula suggested by 

The correlations 
examination of fh^ 

, Even though the sub..., «i«nincantjy reduced 

intercorreJatlons for tJf^. ^''^ <I"ite high ^ correlations aaon* th. 

reJlabiJity for fhf I 5^ ^««t8 .ay be th? . ^f^-^over. disparities hof 
V ^c,r the Academic Profile.^ '^""^^ of higher 1 eve of 

An exa»lnation of ^-k 

to 88 '"5 A^le-ic Jr'f?^,""- ™« range Of KR-20 ! ff* 
he'mJ quarter ij' " for ?he P.n "^^"•""y 

Dlsattenuated corrpia** 
In scale reliability! Provide a means of 

During the winter 



national re.earch ^!.: cow exa. nh., ll " f"*" ^he 

•»1 ror the OLattena:;:.. co'rre'l^Uni^'l^' to 

Because data were n^l»4^a.w% . 

and CJarifying Value.lh.i i * during the Fall ouar?., ,! "'"over, 
those acalM with hTiL ""elatlona in the 2?ntf » • "^oMunlcatlng 

social i«.tit:i*^»;r ?au%:i^ :^ ,"-"-rjy"» 

t*. Winter quarter. Technolo^r*." hlTcorre,°.'tro„'a'rn 

atabir'the'Jellahftif"""' subscales on the rn«D 

During the Fal ";" 5" '""""lea^w"": «J«"-«iy 

tively high (.err ""Jnglhri "^Ing the i"! 

loweat level of relJahiw; , Winter quarter, thl. ,uh.i.T "la- 
fclgheat level, o"re!?ib n,' f"- ''<"• •"■t* quarter, ^""ced the 

reliability (.75 a„d .73 respect^'iyK 

For the Acadeaiic Pmf^ii - 

P-%^\''%r":X'" '-'-e^^^^^^^ .-.«le. .„d the 

tl-e .o,t higMr"r«la "f"' "'''"^" of t"^'!^!"r.?'»"» "ro,. testing 
wrlod. On the other ha„H '"^ **b*ca?« ^Z""^" """^ 

reliable aubscale, "„3''?^ , 'S" ""heaatlc, aub.cale "aj oj "'fr"*' 
aubacale,. "3- relatively l„. corriunin^^lti ':Z\ 

While a visual f>ya»4».«.i 
that both the rnMD ^««»ination of the correlatinn- 

sion. it is'tssfb' :\^roth''^ ^r'^-^'^'-^'i a e^^Sf„»"^^^^ suggests 
Whether there are unin„! diaenslons are presenr r * 

measured by the cniS^ ^^P*^^' the education!? '"'^^^'^ test 

sea ..ere perJor.ff T*" Acade.ic P^on?! ^T'"^^"*^^ ^'^-^ «re belntf 

tions a«ong"subscaies'':na?' to ig/p^^S^'f' -.ponents anafy- 

The presence of a sILi! "^''^ conducted only fo^ ?h ""'^"^ correla- 

--^....enuatTc^r^^^^^^^^^^^^^^ 

fouJd support the vieTthat th '"'^'""^^ ''^ ""^^ thSHL ^rf^'" «"f scales 
two tests. ^^^^ there are several outcoLrh ?^^""Pa^ component 

outcoaes being aeesured by the 

cess/sklurareal^!/thr^S2p ^"'"^^"^"ts analyses for the c„„. . 

four subtables of Table 3 ^ ^"'f* Acade.ic ProfUe arp P'*^" 

structure. m no case dL ^""^^^^ ^"PPO'-t the e^istenr- 

the data analyses eLh "k"* »"nlngful second pJinc!«f? "nidi«ensional 
ties (estimates Of expLi^en''''! t^^^v TLT^^^ in 

are reported r^^t"tL---intt ^te^ '""^^^^ -^a^r^if- 
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Table 3: 

Principal Coaiponents Analysis of the Subscales for the COMP Exaa 

and the Academic Profile 



COMP CONTENT AREAS 





Fall 


Winter 




PATTERN 


COMMU- 


PATTERN 


COMMU- 




LOADING 


NALITY 


LOADING 


NAL I TY 


FSI 


.83 


.69 


.81 


.67 


UST 


.83 


.69 


.84 


.70 


UA 


.74 


.55 


.80 


.64 






COMP PROCESS AREAS 






Fall 




Winter 




PATTERN 


COMMU- 


PATTERN 


COMMU> 




LOADING 


NAL I TY 


LOADING 


NALITY 


COM 


.80 


.65 


.80 


.64 


SP 


.81 


.65 


.81 


.66 


cv 


.77 


.59 


.81 


.66 




ACADEMIC PROFILE 


CONTENT AREAS 




Fall 




Winter 




PATTERN 


COMMU- 


PATTERN 


COMMU- 




LOADING 


NALITY 


LOADING 


NAL I TY 


HUM 


.93 


.87 


.94 


.88 


SS 


.94 


.89 


.95 


.89 


NS 


.93 


.87 


.92 


.84 




ACADEMIC PROFILE 


PROCESS AREAS 




Fall 




Winter 




PATTERN 


COMMU- 


PATTERN 


COMMU- 




LOADING 


NALITY 


LOADING 


NAL I TY 


READ 


.91 


.82 


.92 


.85 


WRIT 


.90 


.81 


.90 


.80 


CT 


.92 


.84 


.92 


.84 


MATH 


.81 


.66 


.81 


.65 
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For the content tubacaies of ths com .... . . 

Fall ouarter produced an .l»envalue ofT^j J^^Srii:''! "« <•«• fo. tfte 
I An eigenvalue of .S3 ra. obtained for the f.f^L *,""* "''^'"'ifi coaponenr 

" f"" '">• Wnter -luarter Ue ^JwJ^lJn 2? ?![ 5 ^or the 

coaponent. mre z oo and .54 reaMctivrr„ ? "»'' ••e»K« prlncloai 

m ented in th. n„t aubt.bl. ln"«?Ii Jj« .n tb. data pr.a- 

•xplaln a aubatantl.l proportion of'lZ'liATncl'^l llTlXc:" " 
on thi;^P ««^"!:o':^,iJ:.SS^"^'!i,::!r-^ -"-"ixm, proce.. .ub.core. 

«:o„^r r=^ - "Trrire-^^^^^^^^^^ 



I 

I 
I 



-.. ex« ajao auggested a unirfi....< , -""•■^'»'n« proceaa aabacarei 

the Fall quarter, elgenv"uea of 1 Si iZJ i^" "or the da" ?r« 

aecond principal coaponenta For ih! ! ? V "fct'laed for tba fi'at .»d 
.ijenyalue, «re ,.9? S3 re.pecu«lv 
had algnlflcant poaltlve loading, on"he ?;..?*".*»'*''- «"•«"" »«b.core. 
coaponent -a. able to explain ''-''I'^^ullt^^^^^^^^^ 

»ulta presented in the third .n^f.h? luarter >he « 

coaponent aodel l. suff c ent ?o e^?.^'""'^" conclu.lon tS^t a^«. 
the Acade.lc Profile. '° ""I'tlon In the content .u',co«. on 

Consistent with resulta fn„ 

niflcant loadings on the f Ir.t'prln'^J^^rnip^^^,;"^"' 'U" 
rnterrelatlo nsjile, Betwen the Testa 

unldl.en*?oM! «rJcJl!".1ie l^.l'lZ *T A"de.lc Profile have a 

any correspondence bet««„ the scoriro? thT,"'' " '» -Aether therl i, 

this question, students' score. o^JJ . * order to an.»rr 
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Table 4* 

Correlatlpn. Bet^en the COKP Ex« .„d Acadclc Profile Subecle, 





FSI 


UST 


UA 


COM 


SP 


CV 


HUM 

SS 

HS 


.34 
.29 
.15 


.54 
.56 
.34 


.49 
.53 
.51 


.51 
.53 
.51 


.56 
.52 
.27 


.36 
.40 
.31 


READ 
WRIT 
CT 
MATH 


.26 
.27 
.26 
.15 


.47 
.45 
.45 

.43 


.43 
.43 
.54 

.50 


.47 
.38 
.52 
.57 


.40 
.46 
.42 
.39 


.36 
.38 
.39 
.21 



the correlation, «o„g oontenr«b,c«« reJeilJ ?i.t th^'^tL."'''"""''" 
a.ong 8ub8c«le» are not for logical conntlrl!!,! f correlation, 
•ubscale Is .ore highly correlated .^Jh n T"^ ! ; ""PJe, the Hu»anltle« 

«lth Ualng the Art, ('L^ S UlUrli the Sftu'rirr.'"'' ^""'"olog, (.84) tlL 
highly correlated with Using the AmJ' ( St i jj.^ i.f^^f,""" ^* ■<«•• 

nology (.34). Baaed on theae reanlt, .f leC- 
the content aubacores of thr^P «« th!'/'i* """^'^ 
l.ted. but there 1. not a on.f?:!on"%%r^^;^^^^^^^^^^^ - ^-"e- 

1. -«'"«rea^^c'urbe«^%%^%~ .ub.c.le. 

"hat they aee. to be .ea.ur"g. BaaTon JVS"'"^'' 

subscalea provided by ACT and ETS «o f lescrlptlona of the 

spends to the Clarifying v°Jirfub2!/^ "i! *"<'"lc Profile corre- 

Co«unlc.tlng .ubacale ?n the coi^ «.i .°" "^""^ «Mltlon. the 

subacale, of Reading, Writ „g t^JM"^* I"""!' 

Matheaatlca aubacairi. MSly^orr!?.'"r 1^' 

aubacales that »ould be exoected t« ^^k! J? ^o-^lcatlng (.57). ivo 
and Solving Proble... H«e™r the corJifaMo-^^rf 

(.42) la lower than the correUtJon of Cr t^i^? """cal" 

(.52) and the correlation of s'lvLg Proble". iJS'S?^?^ '^•-»J<"'"»» 

<loe. not see. to be a one-to-one corrwMndeL! there 

the COUP exa. and the Acade.ic Profll" P^^'o/'Kl 1 1 area, on 

t.o ex™:.%u"" ^he^'.'nT"'?* """"1" ot the 

However, this doe, not Hit the t« Jj^l".'" '"frchangeable . 

the .a.e outcce. It doer.UMeat th^rthf^L. »"* wsuring 

Slightly different .ppr:;?s:r trs^aJSji^rChr^^r^/tc^.r"*''^'' 



Indeed, the 
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presence of a single dlaenslon underlying both exm Mould help explain the 
significant correlations between the natural sciences and the arts, and ^et-een 
critical thinking and cowiunlcatlng. -e«-ween 

After exaalnlng the characteristics of the COW* exaa and the Acadealc 
Profile, it seeas reasonable to conclude that the Acadeaic Profile is slightly 
superior to the COMP exaa in terns of greater Itea difficulty, greater itea 
variance, and greater scale reliability. The practical result is that the 
Acadealc Profile is a better aeasure of individual differences than the COMP 
exaa. The fact that saall score changes can produce very large changes in 
percentile ranks on the COMP exaa also aakes the Acadealc Profile a superior 
instruaent for identifying individual differences. 

Despite these indications of the superiority of the Acadealc Profile the 
doainant finding to this point is that neither the subscales of the COUP exaa 
nor the subscales of the Acadealc Profile are able to differentiate aaong 
unique aspects of student outcoaes. The presence of a single underlying dlaen- 
slon »as conflraed for the content and process/skills subscales of the ?(WP 
!!!!! Vlt ! Acadealc Profile. Moreover, there is every indication that the 
saae dlaenslon underlies both exaas. Based on these findings, it seeas reason- 
able to recoaaend against the use of the subscales of these Instruaents Shile 
a precise identification of the outcoae being aeasured by the twlestJ aust 
Zlil '^r'"' sensitivity of these instruaents to eSucaUon" ex^ri- 

very slallar to what Spearaan (1904) teraed "general intelligence." 

Sensitivity to Educational Effects 

-.n-J*"* sensitivity of an assessaent instruaent to students' educational 
3'i!2nr T «^«»«»^ 1" J"<*«^n» the appropriateness of that 

to dIZ«tr«t! <J"««tions related to educational sensitivity are central 

Mess!ck las""* ^o«»truct validity of an outcoaes aeasure (Jronbach. 1971; 

Profile to'^IL^^r^^"?'**^ sensitivity of the COMP exaa and the Acadealc 
Profile to educational experiences, UTK students' scores on the two tests were 

c^I^u ?iran"crb!n:L"""r'^'"*^' ^-<*-t-aoaent^o^^%!aUo::'weri" 

?hi::^"rtLu:ni iT^TeliTe.^^^^^^^^^ ^^^^^^^^^ 

A careful exaalnatlon of the correlations presented in Table 5 reveals 

crrs'eSo?!:': S'o::]^''^'^*'!-. S^"'^*^^"^ ^^^^ expectatlons'lu^anlttes 
coursework was positively related to the Using the Arts subscale on the COMP 

exaa for both the Fall and Minter quarters (.10 and .10). For bo?h quarters 
natural sciences coursework (.06 and .09) and aatheaatlcs coSrsework TlO and 
^09) were positively related to Using Science and Technolog^ sJallariy 
Seilc'p "TUV" positively related to the HuaaniUes suSscale'if the 

Joinively ^elateS to .hf'v^ V^"'"" »athe.atic8 coursework variable was 
bo?h tHtLa i!^ H . 5! Natural Lcienc^-8 ^^ubscale of the Acadealc Profile for 
couLiori LS I nf ; i;'^ ^^"'^"'•y expectations, social science 

UonHn ?hX?, f ^^ir ^^''^^•t Functioning within Social Institu- 

tions in the Pall (-.08) and a positive correlation with scores on the subscale 

er|c H 
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in the Winter ( 04J c , 



Correlations of COMP pJ^^^^ ^■ 

"ith SelecteJ^rn^"** ^*^«<*«»ic Profile .3.0 
iected Courseirork Hfeaaurw 



FSI 
VST 
VA 
COM 

Sp 

CV 

TOTAL 



HUM 
$S 

m 

READ 
WRIT 
CT 

MATH 
TOTAL 







-.01 


-.08 


.05 


-.18 


.10 


-.09 


-.03 


-.16 


.14 


-.04 


.05 


-.14 


.06 


-.35 



COMP EXAM 



.04 
.06 

-.03 
03 

-.01 
.04 

.03 



WATHE. 

.07 

.30 
-.03 

.16 
-.05 
.02 

.06 



HUNAN 

.12 

.09 
.10 
.02 
.02 

11 

.10 



.04 
-.07 
-.04 
-.07 
-.07 
-.01 

-.03 





«ATHf. 


.00 


.00 


.09 


.09 


.02 


.00 


.05 


.13 


.05 


.33 


.01 


.01 


.05 


.04 



nVMAS 




.02 


-.02 


-.08 


-.05 


-.14 


-.19 


.03 


-.01 


-.04 


-.04 


-.04 


-.09 


-.21 


-.20 


-.07 


-.10 



ACADEMIC PROFILE 



.07 
.32 
.26 
.v^5 
.08 
.14 
29 

.16 



.31 
.03 
.02 
.08 
.07 
.07 
-.02 

.06 



-.04 

-.06 

-.23 

-.05 

-.07 

-.04 

-.29 

.12 



-.09 
-.03 
.19 
-.07 
-.11 
.02 
.27 

.03 



and natural scw quarter. /L '^'"" *o the 11.^ V"' "a'urai 

'•"inking durr„i\"L%"T"""* vari^We^ "he c" !f 
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and tifinter quarters, hunanitles coursework was positively related to Solving 
Probleas (.14 and .12 respectively). 

The relationships identified in Table 5 provide a clear indication that 
neither the COUP exas nor the Acadeaic Profile is particularly sensitive to 
students' educational experiences (at least as they are aeasured by patterns of 
coursework). Indeed, 'the key to enhanced perfornance on both the COMP exaa and 
the Acadeaic Profile is not to be found in a broad ^neral education. Instead, 
the key to iaprove test perforaance is a specialized pattern of course taking 
that deeaphasises social science courseifork and eaphaslzes aatheaatics. natural 
science, and to a lesser extent, huaanities courseirork. 

One explanation for the relationships aaong coursework aeasures and test 
scores is that soae outside variable Is producing spurious relationships. 
Previous research has found that students' entering levels of acadeaic ability 
Influence coursework and test perforaance. The net effect is that entering 
ability distorts the relationship between coursework and test perforaance 
(Pike, in press). 

In order to deteraine if acadeaic ability influences perforaance on the 
CONP exaa and the Acadeaic Profile, students' scores were correlated with their 
entering ACT Assesaaent scores. These results are presented in Table 6. 

An exaaination of the correlations presented in Table 6 reinforces the 
view that entering acadeaic ability is a aajor deteralnant of student perfora- 
ance on outcoaes aeasures like the CONP exaa and the Acadeaic Profile. As the 
data In Table 6 clearly show, all of the subscales on the C(mP exaa and the 
Acadeaic Profile are significantly related to students' ACT Assessaent scores. 
For the Fall quarter, the range of correlations is froa .35 (Using the Arts) to 
.51 (Coaaunicating) on the subscales of the COMP exaa, and froa .59 (Reading) 
to .70 (Matheaatlcs) on the subscales of the Acadeaic Profile. For the Winter 
quarter, the range is froa .39 (Solving Probleas) to .50 (Coaaunicating) on the 
COMP exaa. and froa .56 (Writing) to .67 (Social Sciences) on the Acadeaic 
Profile. ACT Assessaent scores are significantly correlated with total scores 
on the COMP exaa (.57 and .56) and on the Acadeaic Profile (.73 and .69) for 
both the Fall and Winter quarters. 

While the correlations between ACT Assessaent scores and CONP scores are 
soaewhat lower than coaparable correlations for the Acadeaic Profile, this is 
not a cause for celebration. At least part of this difference is due to lower 
levels of reliability for the COMP exaa. 
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Table 6: 

Correlations of COMP Exaa and Acadealc Profile Scores 
with Entering ACT Assessment Scores 



CONP EXAM 



Fall 



Winter 



FSI 

UST 

UA 

COM 

SP 

CV 


.49 

.fio 

.35 
.51 
.40 
.42 


.43 
.47 
.41 
.50 
.39 
.41 


TOTAL 


.57 


.56 




ACADEMIC PROFILE 






Fall 


Winter 


HUM 

SS 

NS 

READ 
MRIT 

CT 

MATH 


.07 
.69 
.69 
.59 
.60 
.69 
.70 


.61 
.67 
.66 
.60 
.56 
.65 
.65 


TOTAL 


.73 


.69 



Conclusion 

Taken as a whole, the present research found very little that would rcco»- 
.nd either the COMP exaa or the Acadealc Profile as outcoses aeasures. In- 
-ed. the one clear finding to energe fro* this research is that both the COMP 
am and the Acade«ic Profile are unldlaenslonal aeasures that are not partlcu- 
riy sensitive to students' general education experiences. As a result 
stltutlons would be unwise to use the subscales of the COMP exan or the 
adeaic Profile to evaluate the coaponents of a currlculua. Use of these 
ales to suggest currlculua changes could even be counterproductive. 

^JiLITa^^/ Acadealc Profile represent measures of 

dividual differences. More specifically, these tests are very powerful 
asures of general acadealc ability. Because of their sensitivity to indlvid- 
1 differences, the COMP exaa and the Acadealc Profile are not appropriate for 
aluatlng the iapact of general education prograas appropriate ror 
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